77 research outputs found
An algorithm for cross-lingual sense-clustering tested in a MT evaluation setting
Unsupervised sense induction methods offer a solution to the
problem of scarcity of semantic resources. These methods
automatically extract semantic information from textual data
and create resources adapted to specific applications and domains of interest. In this paper, we present a clustering algorithm for cross-lingual sense induction which generates
bilingual semantic inventories from parallel corpora. We describe the clustering procedure and the obtained resources. We then proceed to a large-scale evaluation by integrating the resources into a Machine Translation (MT) metric (METEOR). We show that the use of the data-driven sense-cluster inventories leads to better correlation with human judgments of translation quality, compared to precision-based metrics, and to improvements similar to those obtained when a handcrafted semantic resource is used
La place de la désambiguïsation lexicale dans la traduction automatique statistique
Word Sense Disambiguation (WSD) is often omitted in Statistical Machine Translation (SMT) systems, as it is considered unnecessary for lexical selection. The discussion
on the need ofWSD is currently very active. In this article we present the main positions on the subject.We analyze the advantages and weaknesses of the current conception of WSD in SMT, according to which the senses of ambiguous words correspond to their translations in a parallel corpus. Then we present some arguments towards a more thorough analysis of the semantic information induced from parallel corpora and we explain how the results of this analysis could
be exploited for a more flexible and conclusive evaluation of the impact of WSD on SMT
Capturing lexical variation in MT evaluation using automatically built sense-cluster inventories
The strict character of most of the existing Machine Translation (MT) evaluation metrics does not permit them to capture lexical variation in translation. However, a central
issue in MT evaluation is the high correlation that the metrics should have with human judgments of translation quality. In order to achieve a higher correlation, the identification of sense correspondences between the compared translations becomes really important. Given
that most metrics are looking for exact correspondences, the evaluation results are often misleading concerning translation quality. Apart from that, existing metrics do not permit one to make a conclusive estimation of the impact of Word Sense Disambiguation techniques into
MT systems. In this paper, we show how information acquired by an unsupervised semantic analysis method can be used to render MT evaluation more sensitive to lexical semantics. The sense inventories built by this data-driven method are incorporated into METEOR: they replace WordNet for evaluation in English and render METEOR’s synonymy module operable in French. The evaluation results demonstrate that the use of these inventories gives rise to an increase in the number of matches and the correlation with human judgments of translation quality, compared to precision-based metrics
Data-driven Synset Induction and Disambiguation for Wordnet Development
International audienceAutomatic methods for wordnet development in languages other than English generally exploit information found in Princeton WordNet (PWN) and translations extracted from parallel corpora. A common approach consists in preserving the structure of PWN and transferring its content in new languages using alignments, possibly combined with information extracted from multilingual semantic resources. Even if the role of PWN remains central in this process, these automatic methods offer an alternative to the manual elaboration of new wordnets. However, their limited coverage has a strong impact on that of the resulting resources. Following this line of research, we apply a cross-lingual word sense disambiguation method to wordnet development. Our approach exploits the output of a data-driven sense induction method that generates sense clusters in new languages, similar to wordnet synsets, by identifying word senses and relations in parallel corpora. We apply our cross-lingual word sense disambiguation method to the task of enriching a French wordnet resource, the WOLF, and show how it can be efficiently used for increasing its coverage. Although our experiments involve the English-French language pair, the proposed methodology is general enough to be applied to the development of wordnet resources in other languages for which parallel corpora are available. Finally, we show how the disambiguation output can serve to reduce the granularity of new wordnets and the degree of polysemy present in PWN
BERT Knows Punta Cana is not just Beautiful, it's Gorgeous : Ranking Scalar Adjectives with Contextualised Representations
Adjectives like pretty, beautiful and gorgeous describe positive properties of the nouns they modify but with different intensity. These differences are important for natural language understanding and reasoning. We propose a novel BERT-based approach to intensity detection for scalar adjectives. We model intensity by vectors directly derived from contextualised representations and show they can successfully rank scalar adjectives. We evaluate our models both intrinsically, on gold standard datasets, and on an Indirect Question Answering task. Our results demonstrate that BERT encodes rich knowledge about the semantics of scalar adjectives, and is able to provide better quality intensity rankings than static embeddings and previous models with access to dedicated resources.Peer reviewe
Capturing Lexical Variation in MT Evaluation Using Automatically Built Sense-Cluster Inventories
PACLIC 23 / City University of Hong Kong / 3-5 December 200
Towards Faithful Model Explanation in NLP: A Survey
End-to-end neural NLP architectures are notoriously difficult to understand,
which gives rise to numerous efforts towards model explainability in recent
years. An essential principle of model explanation is Faithfulness, i.e., an
explanation should accurately represent the reasoning process behind the
model's prediction. This survey first discusses the definition and evaluation
of Faithfulness, as well as its significance for explainability. We then
introduce the recent advances in faithful explanation by grouping approaches
into five categories: similarity methods, analysis of model-internal
structures, backpropagation-based methods, counterfactual intervention, and
self-explanatory models. Each category will be illustrated with its
representative studies, advantages, and shortcomings. Finally, we discuss all
the above methods in terms of their common virtues and limitations, and reflect
on future work directions towards faithful explainability. For researchers
interested in studying interpretability, this survey will offer an accessible
and comprehensive overview of the area, laying the basis for further
exploration. For users hoping to better understand their own models, this
survey will be an introductory manual helping with choosing the most suitable
explanation method(s).Comment: 62 page
Representation Of Lexical Stylistic Features In Language Models' Embedding Space
The representation space built by pretrained Language Models (LMs) encodes
rich information about words and their relationships (e.g., similarity,
hypernymy/hyponymy, polysemy) as well as abstract semantic notions (e.g.,
intensity). In this paper, we demonstrate that lexical stylistic notions such
as complexity, formality, and figurativeness, can also be identified in this
space. We show that it is possible to derive a vector representation for each
of these stylistic notions, from only a small number of seed text pairs. Using
these vectors, we can characterize new texts in terms of these dimensions using
simple calculations in the corresponding embedding space. We perform
experiments on five datasets and find that static embeddings encode these
features more accurately at the level of words and phrases, whereas
contextualized LMs perform better on longer texts. The lower performance of
contextualized representations at the word level is partially attributable to
the anisotropy of their vector space, which can be corrected through techniques
like standardization to further improve performance.Comment: Accepted at *SEM 202
- …